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Abstract 


Three sections of the same distance education class completed a series of 
Web-based concept map assessments using one of two methods. Open-ended 
maps applied in section 1 led students to conduct more relational thinking 
overall, but variance in map items was very high introducing more sub- 
jectivity in scoring. Pre-selected term mapping applied in sections 2—3 led 
students to correctly classify many concepts and express proper relationships 
compared to an instructor’s map. Identfying expected concept sets from 
the instructor’s maps caused students some difficulty. The high volume of 
readings associated with the task appeared to further this problem. Sttidents 
touted many benefits of mapping, incltiding synthesizing and connecting 
course material, reading more intentionally, and thinking critically. 


Introduction 

T he difficulty in delivering and securing traditional tests at a distance 
is well known (Shyles, 2002). Concept mapping provides distance 
instructors with an alternative assessment tool to determine what 
students know about a relatively narrow domain of knowledge tied to 
required readings or lahs. To complete a concept map, the student is 
either given or extracts key concepts in a domain, groups terms in “like” 
sets, and draws lines between terms to illustrate key relationships. The 
concept map also includes linking words on connecting lines to explain 
relationships (e.g., clear cutting — causes — erosion). Combining concepts 
and linking words creates a “proposition” that may be evaluated for ac- 
curacy (Cardellini, 2004). 

Web-based concept map assessments were integrated in three sec- 
tions of the same graduate teacher education class between fall semesters 
2005 and 2006. The sections were taught entirely at a distance using the 
WebCT Vista (Blackboard, 2007) course management system. Concept 
maps provided an alternative assessment to gauge student understanding 
of course topics. The study helped to determine the most appropriate 
mapping techniques for eliciting and scoring student representations 
using performance and attitudinal measures. 

Concept Map Task Structures andThinking 

Thinking skills may be divided into basic skills and complex processes. 
Presseisen (2001, p. 49) reflects on numerous taxonomies and suggests 
five basic skills in increasing complexity: qualifying or identifying basic 
units and facts, classifying, finding relationships, transforming the known 
to a unique metaphor or analogy, and drawing conclusions such as causes 
and effects. These underlying skills are applied to complex thinking 
processes such as problem solving, decision making, critical and creative 
thinking. 

A number of concept mapping task structures are available that may 
elicit these thinking skills and processes differentially. Highly structured 
maps known as “construct-on-scaffold” or “flll-in-the-structure” only 


require students to define and identify existing knowledge as they fill-in 
the blanks on a provided hierarchical structure (Chang, Sung, & Chen, 
2001; Naveh-Benjamin, McKeachie, Lin, & Neely, 1998). 

More challenging are pre-selected term maps that require the student 
to take a set of given terms, recognize sequences, and deduce relationships 
from the context of course materials. These maps may provide students 
with some structure to start their mapping such as superordinate head- 
ers, or superordinate concepts may be embedded among the pre-selected 
terms, requiring the student to generate the appropriate hierarchical 
structure. 

More challenging still are seeded maps where the student must supple- 
ment provided terms with a specified number of additional terms selected 
on their own. This task increases the challenge by requiring students to 
evaluate information to select important and relevant terms, and to induce 
relationships based on incomplete information. Like pre-selected term 
maps, seeded maps may or may not start off with a partial structure to 
help students integrate new concepts. 

Open-ended maps are perhaps the most challenging, providing no 
terms for the student, only a topic for mapping. Open-ended maps 
provide more opportunities for students to explain and externalize 
their understanding (Ruiz-Primo, Shavelson, Li, & Schultz, 2001), and 
students are more likely to generate original knowledge representations 
through this format. However, the higher-level thinking skills required 
in open-ended mapping (evaluating information, inferring appropriate 
information to represent a domain, inducing relationships) present a 
challenge for even advanced learners. 

In one study, 26 college students were unable to represent arguments 
in their maps that would support integrative statements about theories of 
learning (e.g., “all learning is under the control of reinforcers”) (DeSimone, 
Schmid, & McEwen, 2001, p. 270). Even given some structure for their 
map in the form of integrative statements, students did not spontaneously 
apply higher-level inferencing skill and compile evidence in maps to build 
arguments for the statements. In another study with 17 undergraduates 
using concept maps and shared journals to reflect on the nature of sci- 
ence, students did not revise their initial propositions over the course of 
a semester (Hermann & Young, 2001). 

Active teacher scaffolding may be required to reach higher levels of 
thinking with maps, rather than assuming students will spontaneously 
apply reasoning (DeSimone et ah, 2001). DeSimone et al.’s (2001, p. 
280) students requested ongoing feedback regarding “their apparent 
understanding of the content.” Researchers have developed scaffolds 
that might help students gauge the correctness of their understanding 
in a concept map environment. The MindNet tool includes a concept 
library and map analyzer, allowing learners to review the propositions 
and cluster the maps of other teams, promoting reflection on divergent 
perspectives and discussion among a team regarding their own structure 
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use knowledge... 

make decisions, sotve problems, think critically, evaluate peers' maps. 
\ ^ integrate and use content evidence to justify map structure/reiationships 



cognitive load, learner and scorer, 
amount of content to map or grade 


Figure 1: Thinking skills and processes supported by different mapping tasks. 

(Cheung, 2006). Also, similarity indices allow learners to evaluate how 
well their map compares to peer and expert structures. The Cmap tool 
includes “knowledge soups” for individuals or teams to share propositions 
with others who can evaluate and re-use posted propositions (Institute 
for Human and Machine Cognition, 2007). 

After students create any of the map types discussed in this section 
(fill-in-the-blank, pre-selected, seeded, open-ended), they can use this 
knowledge to engage in complex thinking processes such as decision 
making and problem solving (Tergan, Graber, & Neumann, 2006). For 
example, a teacher might ask students to reflect on the relationships 
between air pollution and asthma in their concept map, and then make 
decisions for appropriate controls. Scaffolding may also be required for 
students to appropriately use the knowledge in their maps for complex 
thinking processes. 

Figure 1 represents the theoretical model created by the researcher 
for this study. In the framework, the thinking skills and processes sug- 
gested by Presseisen (2001) and Tergan et al. (2006), are grounded in the 
context of different concept map task structures. Fill-in and pre-selected 
term maps are suggested to require lower-level thinking skills and less 
cognitive challenge, while seeded and open-ended maps are suggested to 
require both lower- and higher-level thinking skills and more cognitive 
challenge. Teachers can task students with 1) constructing concept maps 
and exercising related thinking skills, and/or 2) using the knowledge in a 
completed concept map to engage in complex thinking processes (problem 
solving, decision making, etc.). 

Using Concept Maps for Assessment 

A variety of methods are available for scoring concept maps, including a 
comparison of the student-created map to an expert or criterion map, a 
computed score based on map elements, or a combination of both (Ruiz- 
Primo & Shavelson, 1996). McClure, Sonak, and Suen (1999) outline 
at least two additional scoring systems: structural, or computing a score 
from higher-level structures, cross-links, and propositions; and relational, 
or computing a score from propositions alone. The relational method is 
touted for its ease of use, mechanical simplicity, and increased reliability, 
allowing scores to be easily defended. 

Others suggest, however, there is too much data in concept maps to 
focus on relationship statements alone (Canas et ah, 2004). The volume 
of data held in concept maps is increasing through new computer tools 
that enhance the static concept map to a more descriptive content map 
with attached external resources (e.g., notes, Web links, digital library 


resources, images) (IHMC, 2007;Tufts University Academic Technology, 
2005). Across map types from pre-selected to open-ended, students can 
use content to elaborate on a structure they have generated or to justify 
their structure on the basis of attached external evidence. 

Scoring maps on the basis of multiple characteristics takes time and 
increases cognitive load for the scorer, particularly for the new content mz.’p 
format with multiple external resources. It is understandable why some 
would limit the scope of mapping assessments to flil-in-the-structure, and 
to no surprise such maps correlate closely with standard multiple choice 
tests (Schau, Mattern, Zeilik, Teague, & Weber, 2001). They should, 
since students are not creating an original structure, but repeating core 
knowledge as on a multiple choice test. In this study, both open-ended 
and pre-selected term maps were investigated as promising task structures 
beyond flll-in maps. Reflections on student performance and teacher scor- 
ing should assist others interested in applying concept map assessments 
to distance teacher education courses. 

Method 

The purpose of this study was to compare two concept mapping methods 
(pre-selected term maps, open-ended maps) and judge their merit as 
alternative assessments for distance teacher education. Research ques- 
tions include: 

1 . How successful are teachers on different mapping tasks in terms 
of accurately representing assigned reading material in visual 
form? 

2. What thinking skills are supported by different mapping 
tasks? 

3. What are teacher reactions to different mapping tasks? 

4. Is one mapping task more appropriate for student assessment 
or for student thinking/learning? 

Participants 

Study participants included 4 1 graduate students enrolled in three sections 
of the same graduate distance education course taught by the researcher 
over three semesters. The graduate course was based on topics associated 
with the integration of technology and related issues in K-12 schools. 
The distance course was delivered asynchronously on the WebCT Vista 
course management system (Blackboard, 2007). Almost all participants 
were full-time, in-service teachers matriculating part-time at the university 
toward a Master’s degree, with only a few non-teaching students in the 
sample. Section 1 included 1 1 students (8 female, 3 male). Sections 2-3 
included 30 students (27 female, 3 male). The grade levels and subject 
areas taught by teacher participants varied widely, however many taught 
high school business education courses. Most of the enrollees had limited 
prior knowledge of concept mapping. Students in sections 2-3 were asked 
on a course pre-survey if they had ever created an electronic concept map 
before, and 76.9% indicated “no” they had not. 

Materials 

Cmap Tools freeware enabled students to develop online concept maps 
and attach external resources to their maps (IHMC, 2007). Each student 
downloaded and installed the program on his or her computer. Cmap is 
practical for distance courses, since it allows instructors to store student 
folders and map templates on servers for remote access, and students 
separated by distance can co-edit shared maps. IHMC’s public Cmap 
servers were utilized for beta testing with section 1 , and the researcher 
installed a Cmap server on campus for use with sections 2-3. 

Students used Cmap to represent concepts from course resources 
and readings. In section 1 , students were required to integrate external 
resources with their maps, but they were not told which resources to 
integrate (e.g., course readings, Web links). In sections 2-3, students 
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were also required to integrate 
external resources with their 
maps, but the resources were 
limited to electronic copies 
of assigned readings placed in 
each student’s map folder. 

Also placed in each stu- 
dent’s map folder were one 
or more map templates. For 
section 1, blank map files 
were added to the students’ 
folders to which the students 
added their own concepts 
and began their open-ended 
mapping. For sections 2-3, 
pre-selected term maps based 
on instructor maps were added 
to student folders. To create 
each instructor map, key terms 
were selected from assigned 
reading material and organized 
around a set of superordinate 
headers (e.g., see Figure 2). 

To create the corresponding 
pre-selected term map for students, the links and 
structure were simply removed from the instructor 
map (see Figure 3). 

Procedures 

Students completed between two and three concept 
maps related to course resources and assigned read- 
ings. In section 1 , open-ended maps were employed 
with no terms provided to students in advance. 

Students were responsible for extracting relevant 
concepts from the course resources and readings, 
organizing and relating them, and integrating 
external resources to illustrate where they found 
depicted relationships. 

In sections 2-3, students were provided with 
approximately 20 pre-selected terms in advance of 
their reading, and they were responsible for organiz- 
ing and relating those terms on their maps. Three or 
four superordinate concepts were listed at the top 
of each map under which students classified their 
terms. Table 1 summarizes assigned map activities by section. Table 1 : Summary of Courses, Students, and Mapping Activities. 
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■igure 3: Student pre-selected term map created from instructor map. 



Figure 2. Instructor map illustrating concept classifications, sets, and propositions. 


Analysis 

A case study design was employed with section 1 and sections 2-3 rep- 
resenting two separate units of analysis. Across sections, three student 
data sources were used: a log of student questions posed to the instructor 
regarding assigned mapping tasks, student responses to an online survey, 
and concept map artifacts. The online survey queried students as to such 
items as the usefulness of the Cmap assignment, how much course content 
should be used in creating a Cmap, whether students preferred to work 
on Cmaps alone or collaboratively, and how well Cmap helped students 
make conceptual connections within and between course sessions. 

The conceptual analysis method of content analysis was applied to 
questions received by the students as well as open-ended survey comments, 
identifying frequently used keywords and sorting text by themes. Numeri- 
cal student survey responses were summarized with descriptive statistics, 
and a multivariate analysis of variance (MAN OVA) was employed to 
determine any differences in response between cases. 


Section 

Enrolled 

Maps Completed and Content 
Coverage 

Task 

Structure 

Section 1, 

11 students 

2 individual maps: 

open- 

Fall 05 


the Internet (2 weeks, 8 articles) 

ended. 



influencing factors on tech 

no terms 



integration (1 week, 4 articles) 

provided 

Sections 2-3, 

15 students 

3 individual maps: 

pre- 

Summer 06, 

each 

hardware (3 weeks, 7 articles) 

selected 

Fall 06 


software (4 weeks, 14 articles) 

terms 



the Internet (3 weeks, 14 articles) 

provided 


Methods for scoring concept maps varied by section. For section 1, 
the number of concepts, correct propositions, and integrated resources 
were tallied. Further, content analysis was applied to individual maps 
to identify the superordinate or higher-level categories students placed 
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Figure 4, Excerpt from student open-ended map. 
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Figure 5. Comparison of excerpts from instructor map (left) and student pre-selected term map (right). 


in their maps (i.e., major themes). 

Categories are often exemplified 
with several specific underlying 
concepts, and thus easy to identify 
on a map. Figure 4 illustrates an ex- 
cerpt from one student’s map about 
influencing factors on technology 
integration. While this student’s 
full map included 41 concepts, if 
it were scored as shown it would 
he credited with 17 relevant con- 
cepts, 19 correct propositions, 
and 6 embedded resources as il- 
lustrated by the square icons just 
below several of the concept boxes 
(e.g., below “administrator roles”). 

Further, this sample map includes 
five superordinate categories, each 
with several underlying concepts, 
starting clockwise with professional 
development, collaboration, bud- 
geting, hardware and connectivity, 
and staffing. 

In an effort to create a scoring 
scenario that was less subjec- 
tive, pre-selected term maps were 
employed for sections 2-3. A 
combination scoring method was 
employed, assigning one point 
for each concept, concept set, and 
proposition on the students’ maps 
that were similar to items on a 
comparison instructor’s map (Ruiz- 
Primo & Shavelson, 1996). 

An excerpt from one instructor 
map is shown in Figure 5 from 
which students could receive five 
points for correctly classifying shown terms in any way under the header, 
and another two points for grouping terms in sets — ^WebQuests, digital 
archives, and bookmark managers with volume of Web information, and 
digital archives with student-centered inquiry projects and digital storytell- 
ing activities. Where three or more concepts were co-Iocated in a set, the 
student had to place all concepts together to receive one point. A half-point 
was assigned if only some of the concepts were placed together. 

On the student map shown in Figure 5, the student received five 
points for classifying all of the expected terms under the header (one 
extra term, design projects, was incorrectly classified), and one point for 
identifying the concept set associated with digital archives. The student 
also received six points for correct propositions. Even though the concept 
“design projects” was incorrectly classified by comparison to the instructor’s 
map, the student’s proposition relating this concept to “WebQuests” was 
accurate. 

Results 

Open-Ended Mapping 

In section 1, students placed an average of 34.7 and 37.3 concepts re- 
spectively in their two open-ended concept maps of the type illustrated 
in Figure 4 with a highly divergent standard deviation of 21.8 and 23.4 
respectively (see Table 2). The number of correctly written propositions 
and unique resources integrated in student maps also had a high overall 
variance. Ffigh variance on open-ended map tasks has been reported by 
others (Shaka & Bitner, 1996). 


Table 2: Section 1 Concept Map Results. 


concept map 

/of 

/ of concepts 

/ of resources 

/ of proposition 


maps 

identified 

attached 

statements 


scored 

Mean 

SD 

Mean 

SD 

Mean 

SD 

internet (2 weeks) 

11 

34.7 

21.8 

20.5 

11.B 

28.4 

13.4 

factors (1 week) 

10 

37.3 

23.4 

IB.B 

11.3 

34.1 

18.8 


Using content analysis, the major categories represented on students’ 
open-ended maps were noted (i.e., the higher-level terms on which they 
placed the most importance). Table 3 presents the superordinate catego- 
ries cited most frequently on open-ended student maps and some of the 
underlying concepts suggested by students. 

The categories on open-ended student maps clearly represented topics 
introduced in course units; thus each student was successful at represent- 
ing at least some of the major unit topics on their maps. Ffowever, since 
most student maps contained only three or four categories among seven 
or eight possible, the data further illustrate considerable divergence in 
the focus of student maps. Students emphasized different topics on their 
maps they found most important or relevant. 

For assessment, it was helpful to analyze all student maps as shown 
in Tables 2-3 before assigning points, to develop a standard indicator of 
performance. The decision to deduct points was admittedly subjective, 
based on a holistic consideration of concepts, unit themes represented, 
propositions, and resources. 
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Table 3: Categories and Underlying Concepts Cited Most Frequently on Internet (n=11) and Influencing Factors Maps (n=10). 


Map 

Major Categories on 
Student Maps 

i of Maps 
Citing 

Underlying Concepts Shared by Students 

internet 

communication tools 

11 

blogs, bulletin boards, listservs, e-mail, podcasts, chat, messaging, conferencing 


copyright 

8 

acceptable use policies, internet resources, fair use, public domain 


collaboration tools 

7 

wikis, blogs, class Web pages, global learning projects — thinkquest 


networks 

7 

LAN, WAN, servers, wireless, modems, DSL, cable, shared tools 


Internet resources for 
teachers 

7 

digital archives, Webquests, government sites, online publications, professional organizations, lesson 
plan sharing 

factors 

funding, budgeting 

9 

budget, business and industry, government, NSF, grants, seed money, collection development plans, 
leasing, fundraising, competing needs, restrictive conditions 


professional development 

8 

on learning theories, on research-based best practice models that are replicable, on teaching/ 
mentoring/coaching skills, for teachers and administrators, collaborative teams, e-learning, in-service 


planning 

6 

school technology/improvement plans, district/state/national technology plans, shared decision 
making, goals, strategies, evaluation, SWOT analysis, STAR Charts, collaborative planning 


standards 

5 

ISTE, 21st Century Skills, No Child Left Behind Act, state technology model, national technology plan 


administrative or leadership 
support 

5 

hiring risk takers, flexibility with time, provides resources and professional development, mentors, 
models, advocates, change agents 


One procedural advantage noted for open-ended mapping was the 
ability for students to customize their maps to address personal interests. 
For the Internet map in particular, a few cases were noted where a student 
who taught a subject such as writing would integrate writing- related 
concepts and external resources. 

Pre-Selected Term Mapping 

Pre-selected term maps of the type illustrated in Figure 3 and scored in 
Figure 5 objectively depicted student understanding of specific topics. 
As shown in Table 4, students correctly classified more than 80% of 
pre-selected terms under the superordinate headers provided for each 
of the three assigned maps. Students also generated between 9 and 14 
correct propositions for each map. No correct number of propositions 
was assumed, but Table 4 indicates the number of propositions on each 
instructor map for general comparison. 

Students experienced the most difficulty identifying specific concept 
sets. Compared to the instructor’s map, students only identified 39.4% 
of the expected concept sets on their first map, and improved slightly to 
5 1 .2% and 57.9% for their second and third maps respectively. The diffi- 
culty students had identifying the same concept sets as the instructor might 
be attributed to the depth of each topic, with between 7 and 14 articles 
assigned for each of three maps. Four students in sections 2-3 commented 
on their post-survey that maps were based on too much content: 

Too many articles to try to find the connections. 

Smaller Cmaps would have been more useful for 
me. 

There may have been too many articles to relate. 

With so many articles, students would sometimes find alternative 
groupings for concepts that were different from the instructor’s map but 
still valid. Five students in sections 2-3 commented on their post-survey 
that alternate conceptions were possible: 

Sometimes an idea was vague to me to tie it to a 
particular reading, or I felt it could be tied more 
than one way. 

I think that sometimes people could make different 
connections. 

Providing pre-selected terms to locate and organize across a selection 
of readings did encourage better reading among students in sections 2-3. 
When asked what was most beneficial about Cmap on their post-survey. 


Table 4: Concept Map Scores for Sections 2-3 Completing Pre-selected 
Term Maps. 


Map 

/of 

Maps 

Scored 

/of 

Super. 

Concepts 

Concepts 

Classified 

Sets 

Identified 


Propositions 

Mean SD 

Mean 

SD 

Mean 

SD 

Hardware 

30 

4 

16.6/19 3.1 

2.4/6 

1.8 

13.6/17 

5.6 




(84.7%) 

(39.4%) 


(80%) 


Software 

28 

4 

16.3/20 2.6 

4.6/9 

2.0 

12.7/17 

3.7 




(81.4%) 

(51.2%) 


(74.8%) 


Internet 

28 

3 

13.8/16 1.8 

5.2/9 

1.3 

9.7/12 

4.2 




(85.9%) 

(57.9%) 


(80.7%) 



seven students indicated Cmap helped them to read more thoroughly 
and to go back and re-read: 

It also forced me to go back and carefully examine 
all the readings — even though I had read them previ- 
ously, I always found something I had sort of skimmed 
over or didn’t remember. 

It also gave me a reason to go back to the readings 
and really internalize the information. 

Required me to REALLY read over the material. A 
person might fake their way through some of the 
discussion, but the Cmap was proof positive that I 
had thoroughly read the material. 

It made me think about the readings on a deeper 
level. 

By having to dissect the readings in order to do the 
CMaps, I really digested the information. 

The results presented in Table 4 illustrate that students were able to 
structure and relate specific concepts from their readings in a manner 
moderately similar to the instructor’s map. The class averages were useful 
to assign points, with students falling below the averages receiving fewer 
points. The standard deviations for concepts classified, sets identified, and 
propositions were lower overall than the standard deviations for similar 
points of analysis in open-ended maps, reducing subjectivity in scoring. 
Proposition writing was the skill that varied most across students who 
completed pre-selected term maps. 
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Student Reactions to Mapping 

Regardless of mapping method, student reactions to Cmap as an alterna- 
tive assessment were positive. When asked to choose between a traditional 
exam and a Cmap assessment on their post-survey, more than 90% of 
students in each section chose Cmap (see Table 5). When given more 
choices, including applied projects and a combination of assessment 
methods, a majority of students in each section still chose Cmap as the 
preferred mode. 

Students were also asked on their post-survey if they agreed or disagreed 
Cmap was a useful course assignment. Using a 5-point Likert scale, 86.2% 
of students from section 1 and 87.2% of students from sections 2-3 
agreed or strongly agreed that Cmap was useful. Among three students 
who verbalized a dislike for Cmap in written survey comments, they still 
believed the tool was educational: 

Personally 1 hate doing them, but they are effective 
tools for learning. Therefore, the least beneficial aspect 
is my personal dislike; however there is no educational 
downside. 

Not so much useful, but it was a new experience. It 
challenged beyond memorization. 

Many students in sections 2-3 indicated that just learning to use 
Cmap itself was beneficial. Some teachers in the course indicated they 
had adopted or would adopt concept mapping as a strategy with their 
own students: 

The Cmap activity was immensely useful because 
it armed us with at least one strategy that we could 
immediately use in our classrooms. 

Since 1 had never done one, it was beneficial to 
experience it as a possible tool to use in my own 
classroom. 

1 loved them! We will be doing these in my class in 
2006-07! 

During this course, 1 have incorporated concept 
mapping into my classes. My students now use 
Inspiration. 

Discussion 

Thinking with Cmaps 

The study framework presented in Figure 1 suggests students who 
construct pre-selected term maps and open-ended maps may engage in 
different thinking skills. Study data suggest this is true. 

When asked what was most beneficial about Cmap on their post-sur- 
vey, students who completed open-ended maps in section 1 were more 
likely to suggest the tool helped them think critically about the course 
readings, perhaps because their task required additional higher-level skills 
tied to evaluating and selecting pertinent information for the map: 

Using Cmap helps to develop critical thinking skills, 
not just memorization skills. 

1 think Cmap helps students with critical thinking. 

[It] provoked thought. 

It did help clarify my thinking, and 1 feel helped me 
better retain the information that 1 read. 

When asked the same question, 41% of students who created pre- 
selected term maps suggested the tool helped them make connections, tie 
information together, link information, or group information. The ability 
to explicate relationships was the thinking skill that students constructing 
pre-selected term maps attributed to Cmap most often. This finding is 


Table 5: Percentage of Students Preferring Different Methods of 
Assessment In Distance Teacher Education Courses 




Section 7 
n=U 

Sections 2-3 
n=30 

If you could choose between a 

Cmap 

90% 

93.1% 

Cmap assessment or a regular 
exam, which would you select? 

Regular 

Exam 

10% 

6.9% 

If you could choose between 

Cmap 

40% 

48.3% 

Cmaps, regular exams, or applied 
projects, which form of assessment 

Regular 

Exam 

0% 

3.4% 

would you select? 

Applied 

Projects 

30% 

13.8% 


Combination 

30% 

34.5% 


also consistent with the study framework which suggests pre-selected term 
maps support recognizing sequences and deducing relationships: 

Making connections among concepts was most 
beneficial. 

It forced me to understand and make connections 
between essential course concepts. 

Cmapping helps to tie information together. 

It allowed me to connect all the articles and ideas 
together for each session. 

When asked if Cmap helped them to make connections within spe- 
cific course sessions and between course sessions, 100% of students in 
section 1 and more than 75% of students across sections 2-3 agreed or 
strongly agreed on a 5-point Likert scale (see Table 6). A slightly smaller 
percentage of students in sections 2-3 agreed that Cmap helped them to 
make connections between sessions of the course. The opposite effect was 
predicted, since students in sections 2-3 prepared maps from 3-4 weeks 
of course material compared to students in section 1 who prepared maps 
from only 1-2 weeks of course material. It may be that the restrictive 
nature of pre-selected term mapping, however, left students in sections 
2-3 with the impression of connecting fewer concepts overall even though 
their maps were based on more material. In fact, they did relate fewer 
concepts overall (see mean number of written propositions from Tables 
2 and 4). One student in section two commented that pre-selected term 
maps may lead students to read for specific information and miss other 
general ideas in the process: 

1 think since we were given topics ahead of time, 
it affected how 1 read the articles. 1 was reading for 
specific information, which can be timely and help- 
ful, but 1 missed some important facts, 1 discovered 
upon re-reading. 

Pre-selected term maps do not necessarily keep students from gaining 
“big picture” understanding as suggested by several students in sections 
2-3: 

It allowed me to see the “big picture” and make con- 
nections with all the concepts. 

It helped me to see the big picture. 

The “big picture” in pre-selected term maps, however, must be deduced 
from the instructor’s terms rather than induced from the student’s own 
evaluation of material at large, which the study framework suggests in 
contrasting deduction in pre-selected term maps with induction in open- 
ended maps. As several students suggested, pre-selected term maps were 
more about deducing the instructor’s model of content than constructing 
their own: 
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Table 6: Percentage of Students Who Agree or Strongly Agree With 
Different Values of Concept Mapping Activities 


The Cmap activity helped me to make Section 1 

connections... n=ll 

Sections 2-3 
n=30 

within a specific session of the course. 100% 

82.8% 

between sessions of the course. 100% 

75.9% 

Table 7: Percentage of Students Agreeing or Strongly Agreeing that 
Different Map Strategies Can Be Useful. 

Piease rate whether you agree/disagree the 
foliowing cMap strategies were or couid be useful: 

Section 1 
n=U 

Sections 2-3 
n=30 

strategy one: developing a focused cMap that covers 
only 1-2 sessions of content 

100% 

79.3% 

strategy two: developing a general cMap that covers 
several sessions of content (e.g., sessions 3-11) 

40% 

24.1% 

strategy three: building a Cmap week by week over 
the whole semester 

57.2% 

69% 

strategy four: developing a Cmap for the whole 
semester, but only at the end of semester 

0% 

13.8% 


Seeing the terms that the instructor considered impor- 
tant for each session [was beneficial] . 

The only problem I had was that there is little room 
for difference of opinion with you. I gathered different 
ideas and perspectives from some of the articles.... 

I would like to be able to add other connections that 
were discussed in the readings. 

Map open for interpretation; however, had to match 
instructor’s thoughts. 

Effects of Content Depth 

On a 5-point Likert scale, students were asked on their post-survey if 
they agreed or disagreed that different concept map strategies varying by 
depth of coverage could be useful. Strategy one referred to developing 
focused Cmaps covering only 1-2 sessions of a course. Although this was 
deemed the most useful strategy by students in both section 1 and sec- 
tions 2-3 (see Table 7), a multivariate analysis of variance (MAN OVA) 
revealed a significant main effect for group, T(l, 33) = 4.4, p < .05, = 

.118. Students in section 1 suggested the strategy would be more useful 
than students in sections 2-3, perhaps because section 1 had already 
experienced some success with focused mapping. 

Overall, students in both section 1 and sections 2-3 rated semester- 
long, iterative maps (strategy three) more useful than maps covering several 
sessions of content (strategy two) . This preference may stem from students 
valuing a long-term, but dynamic and changing map. Or for students in 
sections 2-3, the preference may stem from struggles with strategy rwo- 
-mapping content from several sessions of content. All students rejected 
strategy four — a one-shot, non-iterative map covering an entire semester 
of content and developed at the end of the semester. 

Proclivity for Independent Work 

Students in all three sections worked independently on their maps. In the 
first section, students were told they could collaborate, but only two stu- 
dents chose to work together on their second map. On their post-survey, 
students were asked on a 5-point Likert scale if they agreed of disagreed 
that working alone or working collaboratively could be useful strategies for 
developing concept maps. As shown in Table 8, only 37-40% of students 
agreed or strongly agreed that co-developing Cmaps with other classmates 


Table 8: Percentage of Students Who Agree or Strongly Agree with the 
Usefulness of Individual and Collaborative Concept Mapping 


Please rate whether you agree/disagree the 
following cMap strategies were or could be useful: 

Section 1 
n=ll 

Sections 2-3 
n=30 

developing a Cmap by myself 

100% 

39.3% 

co-developing a Cmap with another classmate 

40% 

37.9% 


could be useful. The idea of working collaboratively was not well received 
by the distance students in this course, perhaps because most worked 
full-time and would have difficulty scheduling group mapping sessions. 
Regardless, the benefits of collaborative mapping are well documented 
and should not be ruled out based on student preferences alone (Danish 
& Enyedy, 2007; DeSimone et ah, 2001). 

With regard to mapping alone, a multivariate analysis of variance 
(MANOVA) revealed a significant main effect for group, T(l, 33) = 
6.6, p < .05, if = .167. Section 1 had a significantly higher percentage 
of students who agreed that developing a Cmap alone could be useful. 
This finding is somewhat unexpected, since one might expect students 
tasked with open-ended mapping and the multiple thinking skills required 
of it to desire more support from peers. The application of pre-selected 
term maps in sections 2-3, however, required students to search through 
numerous articles to find assigned terms. Perhaps the challenge added by 
this need to search for and contextualize specific concepts may have led 
students in sections 2-3 to rate the usefulness of independent mapping 
lower. Students in section 1 may have struggled less and required less 
collaborative support, since they could integrate any concept or resource 
they found important into their open-ended maps. 

Conclusions 

In this study, two different methods of Web-based concept mapping 
were investigated as a form of alternative assessment for distance teacher 
education. Each method was found to have both advantages and disad- 
vantages. 

For open-ended mapping, students in section 1 were led to integrate 
more overall concepts and engage in more relational thinking about those 
concepts. Students constructing open-ended maps were also more likely 
to discuss thinking critically about course material consistent with the 
study framework, which suggests the strategy requires students to evaluate 
information, induce meaning, and generate a new knowledge structure 
from a body of information at large. Open-ended maps were found to be 
more flexible in allowing students to customize and integrate their own 
topics of interest and external resources. It follows that great variance 
was evident in the number of concepts and propositions integrated in 
open-ended maps which may present a problem for graders with more 
divergent information to score. 

With pre-selected term maps, students in sections 2-3 were limited 
to a specific set of concepts and thus engaged in less relational think- 
ing overall compared to the open-ended mapping group. Pre-selected 
mapping led students to recognize sequences and make connections 
among course material, consistent with the study framework. The 
strategy encouraged students to read in order to locate specified terms, 
but focus and thinking were more directed toward deducing the fit of 
these terms within the context of focused sections of text. Even though 
students suggested their pre-selected term maps helped them acquire 
“big picture” understanding, this understanding may be better defined 
as a reconstruction of the instructor’s mental model of important course 
content rather than a student’s personal construction of what they found 
important. This finding has implications for assessment, with pre-selected 
term maps akin to a matching task on a test using instructor-provided 
terms, contrasted with an original essay. Customization of pre-selected 
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term maps is discouraged in order to provide a standardized result that 
can be scored objectively. This manifests in lower overall variance in the 
number of concepts, sets, and propositions, which may be appropriate 
for educators seeking a precise assessment instrument. 

For teacher educators, a recommendation is not entirely appropriate for 
one concept mapping task structure over another. Educators interested in 
fostering relational thinking about a course topic may opt for open-ended 
mapping, particularly if the maps are assigned as a project or activity and 
not a scored assessment. Open-ended mapping can be used for assessment, 
but teachers should realize the subjective nature of the scoring task and 
be comfortable with general content analysis. Educators concerned with 
designing an objective assessment may find that pre-selected term map- 
ping carries more precision. In theory, pre-selected term maps carry less 
cognitive load, but as evidenced by this study, increased content depth 
may increase the mental challenge significantly with students required 
to search through material to locate and contextualize provided terms. 
Thus, content depth should be carefully considered when assigning pre- 
selected term maps. 

Overall, concept mapping was well received by students across all 
sections and has become both a valuable learning and assessment tool in 
this course. Perhaps the greatest indicator of success is the large number 
of teachers in the course who have adopted concept mapping in their 
own teaching after their initial exposure to the strategy. 

Concept maps are still used in this course. Specifically, open-ended 
maps are employed with students working in small groups of three to 
co-construct a body of knowledge and engage in more relational think- 
ing than pre-selected term maps could provide. To manage the resulting 
subjectivity in scoring open-ended maps, an additional task is assigned 
to students whereby they use the knowledge in their map to make a 
decision for adopting some aspect of technology integration (e.g., create 
a concept map comparing handheld computing, one-to-one computing, 
and classroom response computing, and make a decision as to which 
hardware system your school should implement using findings/data 
from your map to justify your decision). It is the students’ decisions and 
rationale for their decisions that are emphasized in scoring, rather than 
the details of their maps. The map is used as a means to an end, rather 
than the end product itself This usage is consistent with the two-part 
study framework that suggests students may both build maps using core 
thinking skills and then use the knowledge in their maps to engage in 
complex thinking processes like decision making. 
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Editor’s Remarks continued from p. 74 


implemented in this project provide promising results for the use of online 
approaches to teaching science content to preservice teachers. 

In “A Comparison ofWeh-hased Concept Mapping Tasks for Alterna- 
tive Assessment in Distance Teacher Education,” Kevin Oliver presents 
an innovative approach to testing in a distance education environment. 
Using three different approaches to concept mapping assessment, Oliver 
presents data on student use of these different concept mapping assessment 
approaches. His results provide a positive picture for the use of concept 
mapping assessment in distance education environments. 

In the final article in this issue, “A Model for Facilitating Field Experi- 
ences in a Technology-Enhanced Model Pedagogical Laboratory,” Yuxin 
Ma, Doug Williams, Louise Prejean, Guolon Lai, and Mary Jane Ford 
describe a laboratory experience that uses technology to provide a practice 
teaching environment for preservice teachers. Among other methods, 
the authors use the Teachers’ Beliefs Regarding Technology Use Survey 
(TBTUS) (Park & Ertmer, 2007), to evaluate the effectiveness of the 
laboratory experience and obtain provocative results. 

It is clear that interest in creating and studying innovation is part of 
the glue that holds our community together, both in terms of our practice 
and research and writing. Connecting with the ITEST community will 
contribute to this shared purpose and strengthen both communities. 


President’s Message continued from p. 75 

Awbrey, “sustainable change is not something that is imposed on the 
culture — it emerges from the culture’s self-examination” (p. 14). 
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